Description

Having ship-shape RPCs is crucial for keeping the odds of missing a transmit as low as possible, which is itself crucial for SVR. It is our suspicion (and telemetry concurs) that our nodes are too lenient on RPCs with respect to their polling failures and that there are gains to be had in booting those out of the alive pool. In particular, our nodes do not detect grey-failures — when RPCs fail polling intermittently under a certain rate.

This changes the behaviour in both directions between the unreachable and alive states:

1. In the alive loop, instead of resetting the failure counter, successful polls now only decrement it. This way, we keep the same behavior for error rates above 5/6=0.8333, where an RPC gets booted after 5 probes (on average), but we now eventually declare unreachable nodes that sustain error rates between that and 1:1, which we'd previously tolerate;
2. In the unreachable loop, we add a polling phase to the journey back into Alive, mirroring the other direction and ensuring a node doesn't make it back into alive in the same condition that got it kicked out in the first place. However, instead of following the same "decaying" counter we opted for a stricter flow where a single poll failure resets the node back to square one (dialing). This new phase/step is governed by a new config property PollSuccessThreshold that, defaulting to 0 matches current behavior and allows for progressive, explicit rollout.

Implements DF-23489.

Requires Dependencies

None.

Resolves Dependencies

None.